1. Field of the Invention
This invention relates to the field of system-level programming for a computer system in which a processor has multiple operating modes.
2. Description of the Related Art
The preferred embodiment of this invention is described in terms of a particular virtual computer system, although the invention also applies to a wide variety of other virtual computer systems and to a wide variety of computer systems that do not involve any virtualization.
Virtualization has brought many advantages to the world of computers. As is well known in the art, a virtual machine (VM) is a software abstraction—a “virtualization”—of an actual physical computer system that runs as a “guest” on an underlying “host” hardware platform. As long as a suitable interface is provided between the VM and the host platform, one advantage is that the operating system (OS) in the guest need not be the same as the OS at the system level in the host. For example, applications that presuppose a Microsoft Windows OS can be run in the VM even though the OS used to handle actual I/O, memory management, etc., on the host might be Linux.
It usually requires less than 10% of the processing capacity of a CPU to run a typical application, although usage may peak briefly for certain operations. Virtualization can more efficiently use processing capacity by allowing more than one VM to run on a single host, effectively multiplying the number of “computers” per “box.” Depending on the implementation, the reduction in performance is negligible, or at least not enough to justify separate, dedicated hardware “boxes” for each user.
Still another advantage is that different VMs can be isolated from and completely transparent to one another. Indeed, the user of a single VM will normally be unaware that he is not using a “real” computer, that is, a system with hardware dedicated exclusively to his use. The existence of the underlying host will also be transparent to the VM software itself.
VMware Virtualization Products
The virtualization products of VMware, Inc., of Palo Alto, Calif. provide all of the advantages described above in that they allow multiple, isolated VMs, which may (but need not) have OSs different from each other's, to run on a common hardware platform. The most popular virtualization products of VMware are designed, configured and compiled for operation on a 32-bit x86 hardware platform. For example, the VMware Workstation, ESX Server and GSX Server virtual machine software products are all designed to operate on the x86 platform. The x86 platform was originally developed by Intel Corporation, and is described in numerous different publications, including the IA-32 Intel Architecture Software Developer's Manual.
FIG. 1 illustrates the main components of a system that supports a virtual machine as implemented in the Workstation product of VMware. As in conventional computer systems, both system hardware 100 and system software 200 are included. The system hardware 100 includes CPU(s) 102, which may be a single processor, or two or more cooperating processors in a known multiprocessor arrangement. In the case of the Workstation product, the CPU(s) 102 are based on the x86 architecture. The CPU(s) 102 include a set of registers 103 and a memory management unit (MMU) 108. The system hardware also includes system memory 104 and one or more disks 106. As is well understood in the field of computer engineering, the system hardware also includes, or is connected to, interrupt-handling circuitry, a clock, etc., which, for the sake of simplicity, are not shown in the figure.
The system software 200 either is or at least includes an operating system (OS) 220, which has drivers 240 as needed for controlling and communicating with various devices 110, and usually with the disk 106 as well. In the case of the Workstation product, the OS 220 is a 32-bit OS, such as a 32-bit Windows OS from Microsoft Corporation or a 32-bit Linux distribution. Conventional applications 260, if included, may be installed to run on the hardware 100 via the system software 200 and any drivers needed to enable communication with devices.
As mentioned above, the virtual machine (VM) 300—also known as a “virtual computer”—is a software implementation of a complete computer system. In the VM, the physical system components of a “real” computer are emulated in software, that is, they are virtualized. Thus, the VM 300 will typically include virtualized (“guest”) system hardware 301, which in turn includes one or more virtual CPUs 302 (VCPU), virtual system memory 304 (VMEM), one or more virtual disks 306 (VDISK), and one or more virtual devices 310 (VDEVICE), all of which are implemented in software to emulate the corresponding components of an actual computer. In the case of the Workstation product, the virtualized system hardware 301 is also based on the x86 platform, and the VCPU 302 is an x86 microprocessor.
The VM's system software 312 includes a guest operating system 320, which is a 32-bit guest OS in the case of the Workstation product. The guest OS 320 may, but need not, simply be a copy of a conventional, commodity OS. The system software 312 also includes drivers 340 (DRVS) as needed, for example, to control the virtual device(s) 310. Of course, most computers are intended to run various applications, and a VM is usually no exception. Consequently, by way of example, FIG. 1 illustrates one or more applications 360 installed to run on the guest OS 320; any number of applications, including none at all, may be loaded for running on the guest OS, limited only by the requirements of the VM.
Note that although the hardware “layer” 301 is a software abstraction of physical components, the VM's system software 312 may be the same as would be loaded into a hardware computer. The modifier “guest” is used here to indicate that the VM, although it acts as a “real” computer from the perspective of a user, is actually just computer code that is executed on the underlying “host” hardware and software platform 100, 200. Thus, for example, I/O to the virtual device 310 is actually carried out by I/O to the hardware device 110, but in a manner transparent to the VM.
If the VM is properly designed, then the applications (or the user of the applications) will not “know” that they are not running directly on “real” hardware. Of course, all of the applications and the components of the VM are instructions and data stored in memory, just as any other software. The concept, design and operation of virtual machines are well known in the field of computer science. FIG. 1 illustrates a single VM 300 merely for the sake of simplicity, to illustrate the structure and operation of that single VM; in many installations, there will be more than one VM installed to run on the common hardware platform; all will have essentially the same general structure, although the individual components need not be identical.
Some interface is usually required between the VM 300 and the underlying “host” hardware 100, which is responsible for actually executing VM-related instructions and transferring data to and from the actual, physical memory 104. One advantageous interface between the VM and the underlying host system is often referred to as a virtual machine monitor (VMM), also known as a virtual machine “manager.” Virtual machine monitors have a long history, dating back to mainframe computer systems in the 1960s. See, for example, Robert P. Goldberg, “Survey of Virtual Machine Research,” IEEE Computer, June 1974, p. 54-45.
A VMM is usually a relatively thin layer of software that runs directly on top of a host, such as the system software 200, or directly on the hardware, and virtualizes the resources of the (or some) hardware platform. The VMM will typically include at least one device emulator 410, which may also form the implementation of the virtual device 310. The interface exported to the respective VM is usually such that the guest OS 320 cannot determine the presence of the VMM. The VMM also usually tracks and either forwards (to the host OS 220) or itself schedules and handles all requests by its VM for machine resources, as well as various faults and interrupts. FIG. 1 therefore illustrates an interrupt (including fault) handler 450 within the VMM. The general features of VMMs are well known and are therefore not discussed in further detail here.
In FIG. 1, a single VMM 400 is shown acting as the interface for the single VM 300. It would also be possible to include the VMM as part of its respective VM, that is, in each virtual system. Although the VMM is usually completely transparent to the VM, the VM and VMM may be viewed as a single module that virtualizes a computer system. The VM and VMM are shown as separate software entities in the figures for the sake of clarity. Moreover, it would also be possible to use a single VMM to act as the interface for more than one VM, although it will in many cases be more difficult to switch between the different contexts of the various VMs (for example, if different VMs use different guest operating systems) than it is simply to include a separate VMM for each VM. This invention works with all such VM/VMM configurations.
In some virtual system configurations, the VMM 400 runs as a software layer between the host system software 200 and the VM 300. In other configurations, such as the one illustrated in FIG. 1, the VMM runs directly on the hardware platform 100 at the same system level as the host OS. In such case, the VMM may use the host OS to perform certain functions, including I/O, by calling (usually through a host API-application program interface) the host drivers 240. In this situation, it is still possible to view the VMM as an additional software layer inserted between the hardware 100 and the guest OS 320. Furthermore, it may in some cases be beneficial to deploy VMMs on top of a thin software layer, a “kernel,” constructed specifically for this purpose.
FIG. 2 illustrates yet another implementation, in which a kernel 700 takes the place of and performs the conventional functions of the host OS. The ESX Server product of VMware has this general structure. Compared with a system in which VMMs run directly on the hardware platform, use of a kernel offers greater modularity and facilitates provision of services that extend across multiple virtual machines (for example, resource management). Compared with the hosted deployment, a kernel may offer greater performance because it can be co-developed with the VMM and be optimized for the characteristics of a workload consisting of VMMs.
FIG. 2 also shows a console OS 220B and a set of one or more applications 260 running thereon. The console OS 220B may be used to initially boot-up the virtual computer system and to initiate the loading of the kernel 700. Once the kernel 700 is loaded, the kernel 700 may take over control of the virtual computer system and operate as the primary OS in the system. The console OS 220B may still be used, however, such as to support an application 260 that provides a user with access to configuration data of the kernel 700 and/or the VMM 400. Thus, a system administrator may use the application 260 to control the operating characteristics of the kernel 700 and/or the VMM 400. For example, the user may specify relative resource allocations for multiple virtual machines. The console OS 220B may also be used to provide interfaces for selected hardware devices 110, while the kernel 700 may interface directly with other hardware devices 110. For example, in the case of the ESX Server, the console OS 220B interfaces with some devices 110, such as a mouse and keyboard, while the kernel 700 interfaces directly with other devices 110, such as a Small Computer System Interface (SCSI) adapter connected to a disk drive. For the devices 110 that the kernel 700 controls, the kernel 700 replaces the corresponding interrupt routines in the console OS 220B with references to the appropriate routines within the kernel 700, so that an interrupt related to a device 110 that is serviced by the kernel 700 is routed to the kernel 700, instead of being handled by the console OS 220B. Context switching between the console OS 220B and the kernel 700 may be substantially the same as for the system of FIG. 1. Thus, this invention may also be used in the system of FIG. 2, to switch between the context of the console OS 220B and the context of the kernel 700.
As used herein, the “host” OS therefore means either the native OS 220 of the underlying physical computer, or whatever system-level software handles actual I/O operations, takes faults and interrupts, etc. for the VM. The invention may be used in all the different configurations described above.
Memory Mapping and Address Terminology
In most modern computers, memory is addressed as units known as “pages,” each of which is identified by a corresponding page number. The most straightforward way for all components in a computer to uniquely identify a memory page would be for them all simply to use a common set of page numbers. This is almost never done, however, for many well-known reasons. Instead, user-level software normally refers to memory pages using one set of identifiers, which is then ultimately mapped to the set actually used by the underlying hardware memory.
When a subsystem requests access to the hardware memory 104, for example, the request is usually issued with a “virtual address,” since the memory space that the subsystem addresses is a construct adopted to allow for much greater generality and flexibility. The request must, however, ultimately be mapped to an address that is issued to the actual hardware memory. This mapping, or translation, is typically specified by the operating system (OS), which includes some form of memory management module 245 included for this purpose. The OS thus converts the “virtual” address (VA), in particular, the virtual page number (VPN) of the request, into a “physical” address (PA), in particular, a physical page number (PPN), that can be applied directly to the hardware. (The VA and PA have a common offset from a base address, so that only the VPN needs to be converted into a corresponding PPN.)
When writing a given word to a virtual address in memory, the processor breaks the virtual address into a virtual page number (higher-order address bits) plus an offset into that page (lower-order address bits). The virtual page number (VPN) is then translated using mappings established by the OS into a physical page number (PPN) based on a page table entry (PTE) for that VPN in the page table associated with the currently active address space. The page table will therefore generally include an entry for every VPN. The actual translation may be accomplished simply by replacing the VPN (the higher order bits of the virtual address) with its PPN mapping, leaving the lower order offset bits the same.
To speed up virtual-to-physical address translation, a hardware structure known as a translation look-aside buffer (TLB) is normally included, for example, as part of the hardware memory management unit (MMU) 108. The TLB contains, among other information, VPN-to-PPN mapping entries at least for VPNs that have been addressed recently or frequently. Rather than searching the entire page table, the TLB is searched first instead. If the current VPN is not found in the TLB, then a “TLB miss” occurs, and the page tables in memory are consulted to find the proper translation, and the TLB is updated to include this translation. After the TLB miss fault is handled, the same memory access is attempted again, and this time, the required VPN-to-PPN mapping is found in the TLB. The OS thus specifies the mapping, but the hardware MMU 108 usually actually performs the conversion of one type of page number to the other. Below, for the sake of simplicity, when it is stated that a software module “maps” page numbers, the existence and operation of a hardware device such as the MMU 108 may be assumed.
The concepts of VPNs and PPNs, as well as the way in which the different page numbering schemes are implemented and used, are described in many standard texts, such as “Computer Organization and Design: The Hardware/Software Interface,” by David A. Patterson and John L. Hennessy, Morgan Kaufmann Publishers, Inc., San Francisco, Calif., 1994, pp. 579-603 (chapter 7.4 “Virtual Memory”). Patterson and Hennessy analogize address translation to finding a book in a library. The VPN is the “title” of the book and the full card catalog is the page table. A catalog card is included for every book in the library and tells the searcher where the book can be found. The TLB is then the “scratch” paper on which the searcher writes down the locations of the specific books he has previously looked up.
An extra level of addressing indirection is typically implemented in virtualized systems in that a VPN issued by an application 360 in the VM 300 is remapped twice in order to determine which page of the hardware memory is intended. A mapping module 345 within the guest OS 320 translates the guest VPN (GVPN) into a corresponding guest PPN (GPPN) in the conventional manner. The guest OS therefore “believes” that it is directly addressing the actual hardware memory, but in fact it is not. Of course, a valid address to the actual hardware memory address must, however, ultimately be used.
An address mapping module 445 in the VMM 400 therefore takes the GPPN issued by the guest OS 320 and maps it to a hardware page number PPN that can be used to address the hardware memory. From the perspective of the guest OS, the GVPN and GPPN are virtual and physical page numbers just as they would be if the guest OS were the only OS in the system. From the perspective of the actual host OS, however, the GPPN is a page number in the virtual address space, that is, a VPN, which is then mapped into the physical memory space of the hardware memory as a PPN. Note that in some literature involving virtualized systems, GVPNs, GPPNs, VPNs and PPNs are sometimes referred to as “VPNs,” “PPNs,” “VPNs” and “MPNs,” respectively, where “MPN” means “machine page number,” that is, the page number used to address the hardware memory. The problem is, though, that “VPN” is then used to mean the virtual page number in both the guest and host contexts, and one must always be aware of the current context to avoid confusion. Regardless of notation, however, the intermediate GPPN→PPN mapping performed by the VMM is transparent to the guest system.
Speed is a critical issue in virtualization—a VM that perfectly emulates the functions of a given computer but that is too slow to perform needed tasks is obviously of little good to a user. Ideally, a VM should operate at the native speed of the underlying host system. In practice, even where only a single VM is installed on the host, it is impossible to run a VM at native speed, if for no other reason than that the instructions that define the VMM must also be executed. Near native speed, is possible, however, in many common applications.
The highest speed for a VM is found in the special case where every VM instruction executes directly on the hardware processor. This would in general not be a good idea, however, because the VM should not be allowed to operate at the greatest privilege level; otherwise, it might alter the instructions or data of the host OS or the VMM itself and cause unpredictable behavior. Moreover, in cross-architectural systems, one or more instructions issued by the VM may not be included in the instruction set of the host processor. Instructions that cannot (or must not) execute directly on the host are typically converted into an instruction stream that can. This conversion process is commonly known as “binary translation.”
U.S. Pat. No. 6,397,242 (Devine, et al., “Virtualization system including a virtual machine monitor for a computer with a segmented architecture”), which is incorporated herein by reference, describes a system in which the VMM includes a mechanism that allows VM instructions to execute directly on the hardware platform whenever possible, but that switches to binary translation when necessary. This allows for the speed of direct execution combined with the security of binary translation.
A virtualization system of course involves more than executing VM instructions—the VMM itself is also a software mechanism defined by instructions and data of its own. For example, the VMM might be a program written in C, compiled to execute on the system hardware platform. At the same time, an application 360 written in a language such as Visual Basic might be running in the VM, whose guest OS may be compiled from a different language.
There must also be some way for the VM to access hardware devices, albeit in a manner transparent to the VM itself. One solution would of course be to include in the VMM all the required drivers and functionality normally found in the host OS 220 to accomplish I/O tasks. Two disadvantages of this solution are increased VMM complexity and duplicated effort—if a new device is added, then its driver would need to be loaded into both the host OS and the VMM. In systems that include a host OS (as opposed to a dedicated kernel such as shown in FIG. 2), a much more efficient method has been implemented in VMware's Workstation product. This method is also illustrated in FIG. 1.
In the system illustrated in FIG. 1, both the host OS and the VMM are installed at system level, meaning that they both run at the greatest privilege level and can therefore independently modify the state of the hardware processor(s). For I/O to at least some devices, however, the VMM may issue requests via the host OS 220. To make this possible, a special driver VMdrv 242 is installed as any other driver within the host OS 220 and exposes a standard API to a user-level application VMapp 500. When the system is in the VMM context, meaning that the VMM is taking exceptions, handling interrupts, etc., but the VMM wishes to use the existing I/O facilities of the host OS, the VMM calls the driver VMdrv 242, which then issues calls to the application VMapp 500, which then carries out the I/O request by calling the appropriate routine in the host OS.
In FIG. 1, the vertical line 600 symbolizes the boundary between the virtualized (VM/VMM) and non-virtualized (host software) “worlds” or “contexts.” The driver VMdrv 242 and application VMapp 500 thus enable communication between the worlds even though the virtualized world is essentially transparent to the host system software 200.
The driver VMdrv 242 and the application VMapp 500 are also used when switching between the virtualized world and the non-virtualized world. This switching function is described in U.S. Pat. No. 6,496,847 (Bugnion, et al., “System and method for virtualizing computer systems”) (“the '847 patent), which is incorporated herein by reference. As described in the '847 patent, switching between the two worlds involves a “total processor switch,” including the saving and restoring of all the registers, segments, floating-point registers, and control registers of the processor.
When the system is in the host context, the host OS 220 schedules the applications 260, along with the application VMapp 500, for execution on the system hardware 100 in a conventional multitasking manner. When the application VMapp 500 is scheduled for execution, the application VMapp 500 calls to the driver VMdrv 242. The driver VMdrv 242 initiates the execution of a switch routine that stores the host context and switches over to the VMM context, restoring a previously stored VMM context. As described in the '847 patent, the switch routine executes from a cross page that begins at the same linear address in both the host context and the VMM context. The same function is also performed when switching between one virtualized world and another. Also, when the VMM is ready to relinquish control of the system back to the host context, the VMM calls the driver VMdrv 242, which again initiates the execution of the switch routine. This time, the switch routine stores the VMM context and switches back to the host context, restoring the previously stored host context.
The VMware virtualization products described above are effective and efficient products for creating and supporting one or more virtual machines having the x86 platform, on top of a hardware platform that is also based on the x86 processor. In the Workstation product, the host OS 220 may be a 32-bit OS, such as a standard Microsoft Windows OS or a Linux distribution, and the guest OS 320 may be the same or a different 32-bit OS. However, because of certain limitations of the 32-bit x86 platform, such as a 4-gigabyte (GB) address range (without physical address extension), the x86 platform is beginning to be replaced by 64-bit processor platforms. For example, Advanced Micro Devices, Inc. (AMD) has introduced a 64-bit processor platform that it refers to as AMD64. Intel Corporation has also introduced a 64-bit processor platform that has substantially the same functionality as the AMD64 platform, which it refers to as Extended Memory 64 Technology (EM64T). Processors based on either the AMD64 technology or Intel's EM64T will be collectively referred to as x86-64 processors. Various other companies have developed, or are developing, 64-bit OSs for running on these 64-bit processors, including a 64-bit version of Microsoft's Windows XP OS and 64-bit versions of Linux.
As individuals, companies, schools, governments and other organizations transition from 32-bit hardware platforms, 32-bit OSs and 32-bit applications toward 64-bit hardware platforms, 64-bit OSs and 64-bit applications, there will be a need and/or a desire to execute 32-bit OSs and/or 32-bit applications on a 64-bit hardware platform. The AMD and Intel architectures provide some capabilities for executing 32-bit OSs and/or 32-bit applications (as well as 16-bit OSs and 16-bit applications) on the 64-bit processors, using different operating modes. Specifically, the x86-64 architecture, for example, includes a long mode and a legacy mode. The long mode requires a 64-bit OS, while the legacy mode is used with 32-bit and 16-bit OSs, along with 32-bit and 16-bit applications. The long mode includes two sub-modes, namely a 64-bit mode and a compatibility mode. The 64-bit mode is used for executing 64-bit applications and the compatibility mode is used for executing 32-bit and 16-bit applications under a 64-bit OS. The 64-bit version of the Microsoft Windows XP OS and 64-bit versions of Linux currently provide support, or are developing support, for executing 32-bit applications and drivers in the compatibility mode of the x86-64 processor. When in the legacy mode, the processor operates substantially the same as an x86 processor, at least from the perspective of system software and applications, including a protected mode, a virtual-8086 mode and a real mode. The operating modes of the x86-64 architecture, along with many other aspects of the processor, are described in detail in the AMD64 Architecture Programmer's Manual (“the AMD64 Manual”). The 64-Bit Extension Technology Software Developer's Guide provides similar information for Intel's platform.
The capabilities provided by the x86-64 architecture for executing 32-bit OSs and 32-bit applications are limited, however. For example, the x86-64 architecture does not provide the capability of executing multiple OSs at the same time. VMware virtualization products, as described above, do allow multiple OSs to execute at the same time, in a time-sharing manner, at least for 32-bit OSs. What would be particularly advantageous then, while individuals and organizations are transitioning toward 64-bit OSs and 64-bit applications, is to provide x86 VMs, running 32-bit OSs, on a 64-bit physical processor running a 64-bit host OS. This invention provides such a virtual computer system. Also, the broader teachings of this invention, such as the teaching of an efficient and effective method for switching between multiple software entities that execute using different operating modes of a processor, can be used in a wide variety of other situations, to solve a wide variety of other problems.